feat(rpc): add OpenTelemetry tracing #863

technicallyty · 2025-11-26T21:18:28Z

Description

adds otel tracers to the RPC package.

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

tackled an existing issue or discussed with a team member
left instructions on how to review the changes
targeted the main branch

swift1337

Big! Left comments

rpc/backend/tracer.go

swift1337 · 2025-11-28T17:16:42Z

rpc/backend/account_info.go

-func (b *Backend) GetCode(address common.Address, blockNrOrHash rpctypes.BlockNumberOrHash) (hexutil.Bytes, error) {
-	blockNum, err := b.BlockNumberFromComet(blockNrOrHash)
+func (b *Backend) GetCode(ctx context.Context, address common.Address, blockNrOrHash rpctypes.BlockNumberOrHash) (bz hexutil.Bytes, err error) {
+	ctx, span := tracer.Start(ctx, "GetCode", trace.WithAttributes(attribute.String("address", address.String()), attribute.String("blockNorHash", unwrapBlockNOrHash(blockNrOrHash))))


I see a lot of repeated code that can be simplified with a small wrapper. Consider the following:

Creating span for existing ctx:

// wrapper would be located here: // import https://github.com/cosmos/evm/trace // func Start(ctx context.Context, name string, attr... Attribute) // trace.Str supports string or Stringable // trace.Block automatically parses block or hash // ... other cosmos/evm specific attrs ctx, span := trace.Start(ctx, "GetCode", trace.Str("address", address), trace.Block("block", blockNrOrHash)) // regular span end defer span.End() // captures err if not nil defer span.EndErr(err)

Creating span for new ctx:

ctx, span := trace.StartNew("GetCode", attrs...) defer span.End()

i like the idea of span.EndErr, doing all the cleanup in one go, i could add this. for the other bits of the wrapper, im indifferent. having to do context.Background for a new span is pretty rare (geth API design weirdness). limiting the wrapper to only attributes could be a footgun later down the line if more StartSpanOptions are added that we may want to utilize. currently there are 2 (WithAttributes, WithTimestamp). if you feel pretty strongly though, i'm open to discussing more

i like the End() and EndErr()

might be good to make a trace wrapper in the sdk?

well span.End is already a thing, and adding EndErr would require me to create a wrapper type around the span.

i have already made a change to make ending a span with errors cleaner: add span EndSpanErr helper function

swift1337 · 2025-11-28T17:23:49Z

rpc/namespaces/ethereum/debug/api.go

 	"github.com/cosmos/cosmos-sdk/server"
 )

+var tracer = otel.Tracer("evm/rpc/namespaces/ethereum/debug")


same approach:

// import github.com/cosmos/evm/trace var tracer = trace.New("evm/rpc/namespaces/ethereum/debug")

swift1337 · 2025-11-28T17:29:45Z

Curious, where do we configure tracing sampling rate here?

technicallyty · 2025-12-01T17:45:46Z

Curious, where do we configure tracing sampling rate here?

can be configured on the otel collector side: https://opentelemetry.io/docs/concepts/sampling/

and can be configured client side via otelconf https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/kitchen-sink.yaml#L280

we have this instantiated automatically in cosmos sdk here: https://github.com/cosmos/cosmos-sdk/blob/cfaffe2b286442a262e506845cdf2d05fb6ed2dc/telemetry/config.go

aljo242 · 2025-12-01T19:53:51Z

@technicallyty conflicts

rpc/backend/comet_to_eth.go

 	resBlock *cmtrpctypes.ResultBlock,
 	blockRes *cmtrpctypes.ResultBlockResults,
 ) []*evmtypes.MsgEthereumTx {
+	ctx, span := tracer.Start(ctx, "EthMsgsFromCometBlock")


To fix the problem, replace the assignment to ctx on line 110 with the blank identifier _. This makes it clear to future maintainers that the new context object returned by tracer.Start is intentionally not used, and prevents misleading anyone into thinking that ctx is meant to change in this function. Only the span variable is needed for proper tracing. No further imports, method definitions, or code changes are necessary outside this line.

rpc/backend/comet_to_eth.go

 // BlockBloom query block bloom filter from block results
-func (b *Backend) BlockBloomFromCometBlock(blockRes *cmtrpctypes.ResultBlockResults) (ethtypes.Bloom, error) {
+func (b *Backend) BlockBloomFromCometBlock(ctx context.Context, blockRes *cmtrpctypes.ResultBlockResults) (result ethtypes.Bloom, err error) {
+	ctx, span := tracer.Start(ctx, "BlockBloomFromCometBlock")


The best way to fix this problem is to avoid assigning the new context at all; instead, assign only the needed span variable and discard the new ctx with the blank identifier _.
Specifically: replace

ctx, span := tracer.Start(ctx, "BlockBloomFromCometBlock")

with

_, span := tracer.Start(ctx, "BlockBloomFromCometBlock")

This makes clear to readers (and to linters) that the context is intentionally ignored and that the assignment to ctx was not necessary.

Only line 343 in rpc/backend/comet_to_eth.go needs to be changed, with no need for import or additional definition changes.

aljo242 · 2025-12-03T21:40:39Z

rpc/backend/account_info.go

-func (b *Backend) GetCode(address common.Address, blockNrOrHash rpctypes.BlockNumberOrHash) (hexutil.Bytes, error) {
-	blockNum, err := b.BlockNumberFromComet(blockNrOrHash)
+func (b *Backend) GetCode(ctx context.Context, address common.Address, blockNrOrHash rpctypes.BlockNumberOrHash) (bz hexutil.Bytes, err error) {
+	ctx, span := tracer.Start(ctx, "GetCode", trace.WithAttributes(attribute.String("address", address.String()), attribute.String("blockNorHash", unwrapBlockNOrHash(blockNrOrHash))))


i like the End() and EndErr()

aljo242 · 2025-12-03T21:41:03Z

rpc/backend/account_info.go


-	ctx := rpctypes.ContextWithHeight(height)
-	clientCtx := b.ClientCtx.WithHeight(height)
+	ctx = rpctypes.ContextWithHeight(height, ctx)


this argument ordering is weird

aljo242 · 2025-12-03T21:50:09Z

rpc/types/block.go

 // 0, it will return an empty context and the gRPC query will use the latest block height for querying.
 // Note that all metadata is processed and removed by the CometBFT layer, so it won't be accessible at gRPC server level.
-func ContextWithHeight(height int64) context.Context {
+func ContextWithHeight(height int64, ctxs ...context.Context) context.Context {


i think it is ok to break here instead of this

make the context height thing cleaner

aljo242 · 2025-12-03T21:51:12Z

rpc/backend/account_info.go

-func (b *Backend) GetCode(address common.Address, blockNrOrHash rpctypes.BlockNumberOrHash) (hexutil.Bytes, error) {
-	blockNum, err := b.BlockNumberFromComet(blockNrOrHash)
+func (b *Backend) GetCode(ctx context.Context, address common.Address, blockNrOrHash rpctypes.BlockNumberOrHash) (bz hexutil.Bytes, err error) {
+	ctx, span := tracer.Start(ctx, "GetCode", trace.WithAttributes(attribute.String("address", address.String()), attribute.String("blockNorHash", unwrapBlockNOrHash(blockNrOrHash))))


might be good to make a trace wrapper in the sdk?

technicallyty added 5 commits November 26, 2025 13:17

tracing wip

60f122c

some fixes

b0af048

bug fixes

af08ce4

remove temp tracer from state_transition

bd48256

more fixes

79b76e4

technicallyty changed the title ~~feat: open telemetry~~ feat(rpc): add OpenTelemetry tracing Nov 27, 2025

swift1337 reviewed Nov 28, 2025

View reviewed changes

put tracer in backend.go

d8519e5

technicallyty added 7 commits December 1, 2025 20:56

dont actually need to side effect import telemetry

2910961

tidy

f3b88ba

Merge remote-tracking branch 'origin/main' into technicallyty/otel

58d3c70

add span EndSpanErr helper function

ced7b37

add helper comment

76946cb

consistent import alias

a7f4677

point gomod to sdk otel branch

b9ff28a

github-advanced-security bot found potential problems Dec 3, 2025

View reviewed changes

technicallyty requested a review from swift1337 December 3, 2025 21:03

aljo242 reviewed Dec 3, 2025

View reviewed changes

technicallyty added 2 commits December 3, 2025 14:40

make the context height thing cleaner

aa145ca

fix

f66f004

technicallyty requested a review from aljo242 December 4, 2025 22:37

@@ -107,7 +107,7 @@
             	resBlock *cmtrpctypes.ResultBlock,
             	blockRes *cmtrpctypes.ResultBlockResults,
             ) []*evmtypes.MsgEthereumTx {
-            	ctx, span := tracer.Start(ctx, "EthMsgsFromCometBlock")
+            	_, span := tracer.Start(ctx, "EthMsgsFromCometBlock")
             	defer span.End()
             	var result []*evmtypes.MsgEthereumTx
             	block := resBlock.Block

@@ -340,7 +340,7 @@
             // BlockBloom query block bloom filter from block results
             func (b *Backend) BlockBloomFromCometBlock(ctx context.Context, blockRes *cmtrpctypes.ResultBlockResults) (result ethtypes.Bloom, err error) {
-            	ctx, span := tracer.Start(ctx, "BlockBloomFromCometBlock")
+            	_, span := tracer.Start(ctx, "BlockBloomFromCometBlock")
             	defer func() { evmtrace.EndSpanErr(span, err) }()
             	for _, event := range blockRes.FinalizeBlockEvents {

feat(rpc): add OpenTelemetry tracing #863

Are you sure you want to change the base?

feat(rpc): add OpenTelemetry tracing #863

Uh oh!

Conversation

technicallyty commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Author Checklist

Uh oh!

swift1337 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

swift1337 Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Creating span for existing ctx:

Uh oh!

swift1337 Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Creating span for new ctx:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

swift1337 commented Nov 28, 2025

Uh oh!

technicallyty commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aljo242 commented Dec 1, 2025

Uh oh!

Check warning

Copilot Autofix

Check warning

Copilot Autofix

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

technicallyty commented Nov 26, 2025 •

edited

Loading

swift1337 Nov 28, 2025 •

edited

Loading

swift1337 Nov 28, 2025 •

edited

Loading

technicallyty commented Dec 1, 2025 •

edited

Loading